<!DOCTYPE html>
<html>

<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
	<meta name="theme-color" content="#33474d">
	<title>失落的乐章</title>
	<link rel="stylesheet" href="/css/style.css" />
	
      <link rel="alternate" href="/atom.xml" title="失落的乐章" type="application/atom+xml">
    
</head>

<body>

	<header class="header">
		<nav class="header__nav">
			
				<a href="/archives" class="header__link">Archive</a>
			
				<a href="/tags" class="header__link">Tags</a>
			
				<a href="/atom.xml" class="header__link">RSS</a>
			
		</nav>
		<h1 class="header__title"><a href="/">失落的乐章</a></h1>
		<h2 class="header__subtitle">技术面前，永远都是学生。</h2>
	</header>

	<main>
		
	<span class="different-posts different-posts_earlier">📖 <a href="/page/33">earlier posts</a> 📖</span>




	<article>
	
		<h1><a href="/2017/10/12/LVS/22. keepalived之vrrp_script总结/">keepalived之vrrp_script总结</a></h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/LVS/">LVS</a> <a class="article__tag-link" href="/tags/keepalived/">keepalived</a>
			</span>
		
	</div>

	

	
		<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;通常情况下，利用keepalived做热备，其中一台设置为master，一台设置为backup。当master出现异常后，backup自动切换为master。当backup成为master后，master恢复正常后会再次抢占成为master，导致不必要的主备切换。因此可以将两台keepalived初始状态均配置为backup，设置不同的优先级，优先级高的设置nopreempt解决异常恢复后再次抢占的问题。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;然而keepalived只能做到对网络故障和keepalived本身的监控，即当出现网络故障或    者keepalived本身出现问题时，进行切换。但是这些还不够，我们还需要监控    keepalived所在服务器上的其他业务进程，根据业务进程的运行状态决定是否需要进行主备切换。这个时候，我们可以通过编写脚本对业务进程进行检测监控。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;例如编写个简单脚本查看haproxy进程是否存活</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#!/bin/bash</span></div><div class="line">count = `ps aux | grep -v grep | grep haproxy | wc -l`</div><div class="line"><span class="keyword">if</span> [ <span class="variable">$count</span> &gt; 0 ]; <span class="keyword">then</span></div><div class="line">    <span class="built_in">exit</span> 0</div><div class="line"><span class="keyword">else</span></div><div class="line">    <span class="built_in">exit</span> 1</div><div class="line"><span class="keyword">fi</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在keepalived的配置文件中增加相应配置项</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div></pre></td><td class="code"><pre><div class="line">vrrp_script checkhaproxy</div><div class="line">&#123;</div><div class="line">    script <span class="string">"/home/check.sh"</span></div><div class="line">    interval 3</div><div class="line">    weight -20</div><div class="line">&#125;</div><div class="line"></div><div class="line">vrrp_instance <span class="built_in">test</span></div><div class="line">&#123;</div><div class="line">    ...</div><div class="line">    </div><div class="line">    track_script</div><div class="line">    &#123;</div><div class="line">        checkhaproxy</div><div class="line">    &#125;</div><div class="line">    </div><div class="line">    ...</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;keepalived会定时执行脚本并对脚本执行的结果进行分析，动态调整vrrp_instance的优先级。</p>
<ul>
<li>如果脚本执行结果为0，并且weight配置的值大于0，则优先级相应的增加</li>
<li>如果脚本执行结果非0，并且weight配置的值小于0，则优先级相应的减少</li>
</ul>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;其他情况，维持原本配置的优先级，即配置文件中priority对应的值。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;这里需要注意的是：</p>
<ul>
<li>优先级不会不断的提高或者降低</li>
<li>可以编写多个检测脚本并为每个检测脚本设置不同的weight</li>
<li>不管提高优先级还是降低优先级，最终优先级的范围是在[1,254]，不会出现优先级小于等于0或者优先级大于等于255的情况</li>
</ul>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;这样可以做到利用脚本检测业务进程的状态，并动态调整优先级从而实现主备切换。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;但是利用该方式会存在一个问题，例如：A,B两台keepalived</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;A的配置大概为：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line">vrrp_script checkhaproxy</div><div class="line">&#123;</div><div class="line">    script <span class="string">"/etc/check.sh"</span></div><div class="line">    interval 3</div><div class="line">    weight -20</div><div class="line"></div><div class="line">&#125;</div><div class="line"></div><div class="line">vrrp_instance <span class="built_in">test</span></div><div class="line">&#123;</div><div class="line">    ....</div><div class="line">    </div><div class="line">    state backup</div><div class="line">    priority 80</div><div class="line">    nopreempt</div><div class="line"></div><div class="line">    track_script</div><div class="line">    &#123;</div><div class="line">        checkhaproxy</div><div class="line">    &#125;</div><div class="line"></div><div class="line">    ....</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;B的配置大概为：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line">vrrp_script checkhaproxy</div><div class="line">&#123;</div><div class="line">    script <span class="string">"/etc/check.sh"</span></div><div class="line">    interval 3</div><div class="line">    weight -20</div><div class="line">&#125;</div><div class="line"></div><div class="line">vrrp_instance <span class="built_in">test</span></div><div class="line">&#123;</div><div class="line">    ....</div><div class="line">    </div><div class="line">    state backup</div><div class="line">    priority 70</div><div class="line"></div><div class="line">    track_script</div><div class="line">    &#123;</div><div class="line">        checkhaproxy</div><div class="line">    &#125;</div><div class="line"></div><div class="line">    ....</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;A,B同时启动后，由于A的优先级较高，因此通过选举会成为master。当A上的业务进程出现问题时，优先级会降低到60。此时B收到优先级比自己低的vrrp广播包时，将切换为master状态。那么当B上的业务出现问题时，优先级降低到50，尽管A的优先级比B的要    高，但是由于设置了nopreempt，A不会再抢占成为master状态。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;所以，可以在检测脚本中增加杀掉keepalived进程（或者停用keepalived服务）的方式，做到业务进程出现问题时完成主备切换。</p>

	

	

</article>




	<article>
	
		<h1><a href="/2017/10/12/LVS/23. 阿里云下配置keepalive，利用HAVIP实现HA/">阿里云下配置keepalive，利用HAVIP实现HA</a></h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/LVS/">LVS</a> <a class="article__tag-link" href="/tags/keepalived/">keepalived</a>
			</span>
		
	</div>

	

	
		<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;摘要： 包括阿里云在内的很多云环境，因为不支持浮动IP广受诟病。目前阿里云在VPC网络下发布了HAVIP，能够实现arp宣告IP。这样也就让自己搭建HA成为了可能，有幸拿到了内测权限体验了一下。(classical网络依然不支持) 测试环境： 1 2 3 4 5 6 VPC：192.</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;包括阿里云在内的很多云环境，因为不支持浮动IP广受诟病。目前阿里云在VPC网络下发布了HAVIP，能够实现arp宣告IP。这样也就让自己搭建HA成为了可能，有幸拿到了内测权限体验了一下。(classical网络依然不支持)<br>测试环境：</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/%E9%98%BF%E9%87%8C%E4%BA%91%E4%B8%8B%E9%85%8D%E7%BD%AEkeepalive%EF%BC%8C%E5%88%A9%E7%94%A8HAVIP%E5%AE%9E%E7%8E%B0HA/01.jpeg?raw=true" alt=""></figure></p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"> VPC：192.168.1.0/24 </div><div class="line"> ECS: </div><div class="line"> nginx1：192.168.1.1 </div><div class="line"> nginx2:192.168.1.2 </div><div class="line"> HAVIP:192.168.1.3 </div><div class="line">绑定到havip的公网EIP：121.43.187.37</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;配置完毕后的拓扑如下：</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/%E9%98%BF%E9%87%8C%E4%BA%91%E4%B8%8B%E9%85%8D%E7%BD%AEkeepalive%EF%BC%8C%E5%88%A9%E7%94%A8HAVIP%E5%AE%9E%E7%8E%B0HA/02.jpeg?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;环境搭建完毕后，登陆主备ECS服务器，分别配置nginx+keepalived</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@Nginx1 ~]<span class="comment"># yum install nginx keepalived -y</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;MASTER服务器（nginx1）配置文件/etc/keepalived/keepalived.conf内容以及解释如下：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div></pre></td><td class="code"><pre><div class="line">! Configuration File <span class="keyword">for</span> keepalived </div><div class="line"> <span class="comment">#配置global_defs，主要用于标示机器，以及故障时通知</span></div><div class="line">  global_defs &#123;  </div><div class="line">   router_id Nginx1 </div><div class="line"> &#125;</div><div class="line"> <span class="comment">#配置vrrp_script，主要用于健康检查，以及检查失败后执行的动作。</span></div><div class="line">  vrrp_script chk_nginx &#123; </div><div class="line"> <span class="comment">#健康检查脚本，当脚本返回值不为0时认为失败</span></div><div class="line">   script <span class="string">"/etc/keepalived/ck_ng.sh"</span> </div><div class="line"> <span class="comment">#检查频率，以下配置每2秒检查1次</span></div><div class="line">   interval 2 </div><div class="line"> <span class="comment">#当检查失败后，将vrrp_instance的priority减小5</span></div><div class="line">   weight -5 </div><div class="line"> <span class="comment">#连续监测失败3次，才认为真的健康检查失败。并调整优先级</span></div><div class="line">   fall 3 </div><div class="line"> <span class="comment">#连续监测2次成功，就认为成功。但不调整优先级</span></div><div class="line">   rise 2 </div><div class="line"> &#125;</div><div class="line">  </div><div class="line"> <span class="comment">#定义对外提供服务的VIP vrrp_instance配置</span></div><div class="line">  vrrp_instance VI_1 &#123; </div><div class="line"> <span class="comment">#指定vrrp_instance的初始状态，是MASTER还是BackUP主要还是看优先级。</span></div><div class="line">   state MASTER </div><div class="line"> <span class="comment">#指定vrrp_instance绑定的网卡，最终会通过指定的网卡宣告VIP</span></div><div class="line">   interface eth0 </div><div class="line"> <span class="comment">#发送心跳包的源IP，可使用绑定的网卡IP，也可以使用本服务器上的其他IP</span></div><div class="line">   mcast_src_ip 192.168.1.1 </div><div class="line"> <span class="comment">#相当于VRID，用于在一个网内区分组播，需要组播域内内唯一。</span></div><div class="line">   virtual_router_id 55 </div><div class="line"> <span class="comment">#本机的优先级，VRID相同的机器中，优先级最高的会被选举为MASTER</span></div><div class="line">   priority 101 </div><div class="line"> <span class="comment">#心跳间隔，下面配置，MASTER会每隔1秒发送一个报文高职组内其他机器，自己还活着。</span></div><div class="line">   advert_int 1 </div><div class="line"> <span class="comment">#定义主从的验证方式以及密码，一般使用PASS(最长8位，超过了只会识别前8位作为密码)</span></div><div class="line">   authentication &#123; </div><div class="line">   auth_type PASS </div><div class="line">   auth_pass aliyun </div><div class="line">   &#125; </div><div class="line"> <span class="comment">#VIP,在阿里云下就是刚才创建的HAVIP</span></div><div class="line">   virtual_ipaddress &#123; </div><div class="line">   192.168.1.4 </div><div class="line">   &#125; </div><div class="line"> <span class="comment">#本vrrp_instance所引用的脚本配置，名称就是vrrp_script 定义的容器名</span></div><div class="line">   track_script &#123; </div><div class="line">   chk_nginx</div><div class="line">   &#125; </div><div class="line"> &#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;BACKUP服务器（nginx2）的配置需要修改：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">state MASTER改为 state BACKUP </div><div class="line">mcast_src_ip 192.168.1.1改为backup服务器实际的IP mcast_src_ip 192.168.1.2 </div><div class="line">priority 101改小一些，比如 priority 100</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;其它保持一致即可</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;为了实现nginx服务异常的时候能够自动切换，需要自己写一个脚本，脚本没有硬性的要求，能够实现目标即可，这里 监控nginx进程数为例：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line">vim /etc/keepalived/ck_ng.sh</div><div class="line"></div><div class="line"> <span class="comment">#!/bin/bash</span></div><div class="line"> <span class="comment">#检查nginx进程是否存在</span></div><div class="line">  count=$(ps -C nginx --no-heading|wc -l) </div><div class="line"> <span class="comment">#进程数等于0的时候</span></div><div class="line">  <span class="keyword">if</span> [ <span class="string">"<span class="variable">$&#123;count&#125;</span>"</span> = <span class="string">"0"</span> ]; <span class="keyword">then</span> </div><div class="line"> <span class="comment">#尝试启动一次nginx，停止2秒后再次检测</span></div><div class="line">   service nginx start </div><div class="line">   sleep 2 </div><div class="line">   count=$(ps -C nginx --no-heading|wc -l) </div><div class="line">   <span class="keyword">if</span> [ <span class="string">"<span class="variable">$&#123;count&#125;</span>"</span> = <span class="string">"0"</span> ]; <span class="keyword">then</span> </div><div class="line"> <span class="comment">#如果启动没成功，就杀掉keepalive触发主备切换</span></div><div class="line">   /etc/init.d/keepalived stop </div><div class="line">   <span class="keyword">fi</span> </div><div class="line"> <span class="keyword">fi</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;增加脚本的执行权限</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">chmod +x vim /etc/keepalived/ck_ng.sh</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;分别启动nginx1和nginx2的nginx&amp;keepalived服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">[root@Nginx1 ~]<span class="comment"># /etc/init.d/nginx start;/etc/init.d/keepalived start </span></div><div class="line">Starting nginx: [ OK ] </div><div class="line">Starting keepalived: [ OK ]</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;测试：</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;NGINX1/192.168.1.1的priority 为101，NGINX2/192.168.1.2的priority为100，这时候访问HAVIP绑定的EIP：<a href="http://121.43.187.37/可以看到访问到了服务器NGINX1,如下：" target="_blank" rel="external">http://121.43.187.37/可以看到访问到了服务器NGINX1,如下：</a></p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/%E9%98%BF%E9%87%8C%E4%BA%91%E4%B8%8B%E9%85%8D%E7%BD%AEkeepalive%EF%BC%8C%E5%88%A9%E7%94%A8HAVIP%E5%AE%9E%E7%8E%B0HA/03.jpeg?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;到HAVIP控制台查看，192.168.1.1的服务器为主服务器</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/%E9%98%BF%E9%87%8C%E4%BA%91%E4%B8%8B%E9%85%8D%E7%BD%AEkeepalive%EF%BC%8C%E5%88%A9%E7%94%A8HAVIP%E5%AE%9E%E7%8E%B0HA/04.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;这时候我们KILL掉nginx1服务器的nginx服务</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;查看日志，发送了移除VIP的报文：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">Oct 23 17:20:01 iZ239aqzdi7Z Keepalived_vrrp[25019]: VRRP_Instance(VI_1) sending 0 priority </div><div class="line">Oct 23 17:20:01 iZ239aqzdi7Z Keepalived_vrrp[25019]: VRRP_Instance(VI_1) removing protocol VIPs. </div><div class="line">Oct 23 17:20:01 iZ239aqzdi7Z Keepalived_healthcheckers[25018]: Netlink reflector reports IP 192.168.1.3 removed</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;这时候访问<a href="http://121.43.187.37/，可以看到访问自动切换到了NGINX2" target="_blank" rel="external">http://121.43.187.37/，可以看到访问自动切换到了NGINX2</a></p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/%E9%98%BF%E9%87%8C%E4%BA%91%E4%B8%8B%E9%85%8D%E7%BD%AEkeepalive%EF%BC%8C%E5%88%A9%E7%94%A8HAVIP%E5%AE%9E%E7%8E%B0HA/05.jpeg?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;到HAVIP控制台查看，192.168.1.2的服务器为主服务器</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/%E9%98%BF%E9%87%8C%E4%BA%91%E4%B8%8B%E9%85%8D%E7%BD%AEkeepalive%EF%BC%8C%E5%88%A9%E7%94%A8HAVIP%E5%AE%9E%E7%8E%B0HA/06.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;重新启动nginx1的nginx和keepalive服务</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;查看日志可以看到keepalive重新发送了IP宣告的报文</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">Oct 23 17:22:14 iZ239aqzdi7Z Keepalived_vrrp[25610]: VRRP_Instance(VI_1) Entering MASTER STATE </div><div class="line">Oct 23 17:22:14 iZ239aqzdi7Z Keepalived_vrrp[25610]: VRRP_Instance(VI_1) setting protocol VIPs. </div><div class="line">Oct 23 17:22:14 iZ239aqzdi7Z Keepalived_vrrp[25610]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 <span class="keyword">for</span> 192.168.1.3 </div><div class="line">Oct 23 17:22:14 iZ239aqzdi7Z Keepalived_healthcheckers[25609]: Netlink reflector reports IP 192.168.1.3 added </div><div class="line">Oct 23 17:22:19 iZ239aqzdi7Z Keepalived_vrrp[25610]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 <span class="keyword">for</span> 192.168.1.3</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;重新访问<a href="http://121.43.187.37/测试，重新访问到了服务器NGINX1，到HAVIP控制台查看，192.168.1.1的服务器重新夺回了控制权，成为了为主服务器" target="_blank" rel="external">http://121.43.187.37/测试，重新访问到了服务器NGINX1，到HAVIP控制台查看，192.168.1.1的服务器重新夺回了控制权，成为了为主服务器</a></p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/%E9%98%BF%E9%87%8C%E4%BA%91%E4%B8%8B%E9%85%8D%E7%BD%AEkeepalive%EF%BC%8C%E5%88%A9%E7%94%A8HAVIP%E5%AE%9E%E7%8E%B0HA/07.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;这样就实现了阿里云环境下的HA切换。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;实际测试，阿里云的多个HAVIP可以绑定到同样的两台机器，可以配置多组        vrrp_instance来实现双主。或者两台服务器同时为两个业务服务，避免资源浪费。同时能够做到主备</p>

	

	

</article>




	<article>
	
		<h1><a href="/2017/10/12/LVS/21. Keepalived配置及典型应用案例/">Keepalived配置及典型应用案例</a></h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/LVS/">LVS</a> <a class="article__tag-link" href="/tags/keepalived/">keepalived</a>
			</span>
		
	</div>

	

	
		<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;使用源码先安装keepalived 1.2.6</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="built_in">cd</span> /usr/<span class="built_in">local</span>/src</div><div class="line">wget  http://www.keepalived.org/software/keepalived-1.2.6.tar.gz</div><div class="line">tar zxf keepalived-1.2.6.tar.gz </div><div class="line"><span class="built_in">cd</span> keepalived-1.2.6</div><div class="line">./configure --prefix=/usr/<span class="built_in">local</span>/keepalived </div><div class="line">make</div><div class="line">make install</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;所设keepalived安装到/usr/local/keepalived, 则所有配置文件均位于此目录之下。 我一直没搞明白一个问题，Linux默认总是将程序安装到/usr/local目录下，所有程序共享了/usr/local/sbin目录。这对一些人的操作习惯是有影响的。为何不考虑使用每个软件一个独立的目录呢？</p>
<ol>
<li>建立服务启动脚本，以便使用service命令控制之 </li>
</ol>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">cp /usr/<span class="built_in">local</span>/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived</div><div class="line">chmod +x /etc/init.d/keepalived</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;因为我们使用非默认路径（/usr/local）安装keepalived, 故需要修改几处路径，以保证keepalived能正常启动, 需要修改的文件如下：</p>
<ol>
<li>修改/etc/init.d/keepalived, 寻找大约15行左右的. /etc/sysconfig/keepalived, 修改为： .  /usr/local/keepalived/etc/sysconfig/keepalived, 即指向正确的文件位置</li>
</ol>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;同时在上述行下添加以下内容（将keepavlied主程序所在路径导入到环境变量PATH中）：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">PATH=<span class="string">"<span class="variable">$PATH</span>:/usr/local/keepalived/sbin"</span></div><div class="line"><span class="built_in">export</span> PATH</div></pre></td></tr></table></figure>
<ol>
<li>修改/usr/local/keepalived/etc/sysconfig/keepalived文件，设置正确的服务启动参数 </li>
</ol>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">KEEPALIVED_OPTIONS=<span class="string">"-D -f /usr/local/keepalived/etc/keepalived/keepalived.conf"</span></div></pre></td></tr></table></figure>
<ol>
<li>经过以上修改，keepalived基本安装即可完成，启动测试之： </li>
</ol>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">service keepalived restart</div></pre></td></tr></table></figure>
<ol>
<li>切勿忘记将此服务设置为开机启动</li>
</ol>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">chkconfig keepalived on</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;默认的配置文件中，指定了两个数个虚拟IP : 192.168.200.16  192.168.200.17  192.168.200.18</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;可使用ip addr命令验证之。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;以上实验只需要一台主机，因为当前节点被指定为主节点，且没有收到其它节点的VRRP组播信息，故自动绑定了虚拟IP。</p>
<h2 id="参考案例一：主-备模式"><a href="#参考案例一：主-备模式" class="headerlink" title="参考案例一：主-备模式"></a>参考案例一：主-备模式</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在这种模式下，虚拟IP在某时刻只能属于某一个节点，另一个节点作为备用节点存在。当主节点不可用时，备用节点接管虚拟IP，提供正常服务。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;节点A 192.168.0.11 (主节点), 节点B 192.168.0.12(备用节点)  虚拟IP(对外提供服务的IP 192.168.0.200 </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;要求默认情况下由节点A提供服务，当节点A不可用时，由节点B提供服务(即虚拟IP漂移至节点B)。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;节点A上的配置文件/usr/local/keepalived/etc/keepalived/keepalived.conf</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line">global_defs &#123;</div><div class="line">   notification_email &#123;</div><div class="line">     root@localhost</div><div class="line">   &#125;</div><div class="line">   notification_email_from root@<span class="built_in">local</span> host</div><div class="line">   smtp_server localhost</div><div class="line">   smtp_connect_timeout 30</div><div class="line">   router_id  NodeA</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;默认的配置文件中，使用第三方smtp服务器，但这在现实中几乎没有意义（需要验证的原因），我们将其指定为localhost, 将通知信息的发送交给本地sendmail服务处理。查阅说明文档得知route_id配置是为了标识当前节点，我将其设置为NodeA。当然两个节点的此项设置可相同，也可不相同。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line">vrrp_instance VI_1 &#123;</div><div class="line">    state MASTER   <span class="comment">#指定A节点为主节点 备用节点上设置为BACKUP即可</span></div><div class="line">    interface eth0   <span class="comment">#绑定虚拟IP的网络接口</span></div><div class="line">    virtual_router_id 51  <span class="comment">#VRRP组名，两个节点的设置必须一样，以指明各个节点属于同一VRRP组</span></div><div class="line">    priority 100   <span class="comment">#主节点的优先级（1-254之间），备用节点必须比主节点优先级低</span></div><div class="line">    advert_int 1   <span class="comment">#组播信息发送间隔，两个节点设置必须一样</span></div><div class="line">    authentication &#123;   <span class="comment">#设置验证信息，两个节点必须一致</span></div><div class="line">        auth_type PASS</div><div class="line">        auth_pass 1111</div><div class="line">    &#125;</div><div class="line">    virtual_ipaddress &#123;   <span class="comment">#指定虚拟IP, 两个节点设置必须一样</span></div><div class="line">        192.168.200.16/24</div><div class="line">        192.168.200.17 /24</div><div class="line">        192.168.200.18 /24</div><div class="line">    &#125;</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;默认的配置文件中，竟然没有子网掩码，从而导致使用了默认子网掩码255.255.255.255，如果导致无法从其它机器访问虚拟IP（keepalived虚拟IP无法ping通）。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;按同样的方法配置节点B并修改配置文件，可将A节点的配置文件复制到B节点，并修改以下几项： </p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">router_id  NodeB</div><div class="line">state   BACKUP</div><div class="line">priority   99</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;其它项不必修改。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;测试及验证：拔掉节点A的网线，就发现虚拟IP已经绑定到节点B上，再恢复A节点的网线，虚拟IP又绑定回节点A之上。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;但是这种方式存在恼裂的可能，即两个节点实际都处于正常工作状态，但是无法接收到彼此的组播通知，这时两个节点均强行绑定虚拟IP，导致不可预料的后果。 </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;这时就需要设置仲裁，即每个节点必须判断自身的状态（应用服务状态及自身网络状态），要实现这两点可使用自定义shell脚本实现，通过周期性地检查自身应用服务状态，并不断ping网关（或其它可靠的参考IP）均可。当自身服务异常、或无法ping通网关，则认为自身出现故障，就应该移除掉虚拟IP(停止keepalived服务即可）。主要借助keepalived提供的vrrp_script及track_script实现：</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在keepalived的配置文件最前面加入以下代码，定义一个跟踪脚本： </p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">vrrp_script check_local &#123; <span class="comment">#定义一个名称为check_local的检查脚本</span></div><div class="line">    script <span class="string">"/usr/local/keepalived/bin/check_local.sh"</span> <span class="comment">#shell脚本的路径</span></div><div class="line">    interval 5  <span class="comment">#运行间隔</span></div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;再在vrrp_instance配置中加入以下代码使用上面定义的检测脚本：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">track_script &#123;</div><div class="line">check_local</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;我们在/usr/local/keepalived/bin/check_local.sh定义的检测规则是：</p>
<ol>
<li>自身web服务故障（超时，http返回状态不是200）</li>
<li>无法ping通网关</li>
<li>产生以上任何一个问题，均应该移除本机的虚拟IP(停止keepalived实例即可)</li>
</ol>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;但这里有个小问题，如果本机或是网关偶尔出现一次故障，那么我们不能认为是服务故障。更好的做法是如果连续N次检测本机服务不正常或连接N次无法ping通网关，才认为是故障产生，才需要进行故障转移。另一方面，如果脚本检测到故障产生，并停止掉了keepalived服务，那么当故障恢复后，keepalived是无法自动恢复的。我觉得利用独立的脚本以秒级的间隔检查自身服务及网关连接性，再根据故障情况控制keepalived的运行或是停止。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;这里提供一个思路，具体脚本内容请大家根据自己的需要编写即可。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;如下图的案例：</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/Keepalived%E9%85%8D%E7%BD%AE%E5%8F%8A%E5%85%B8%E5%9E%8B%E5%BA%94%E7%94%A8%E6%A1%88%E4%BE%8B/01.png?raw=true" alt=""></figure> </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;假设节点A和B组成主备关系，A为备用节点，B为主节点，那么当在图标红叉位置发生网络故障时，节点A接收不到节点B的组播通知，将抢占虚拟IP。这时出现的后果就是节点Ａ和节点B均拥有虚拟IP，就可能导致了脑裂。所以我们以网关IP连通性作为参考，可避免此问题产生。</p>

	

	

</article>




	<article>
	
		<h1><a href="/2017/10/12/LVS/20.  haproxy+keepalived实现高可用负载均衡/">haproxy+keepalived实现高可用负载均衡</a></h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/LVS/">LVS</a> <a class="article__tag-link" href="/tags/keepalived/">keepalived</a>
			</span>
		
	</div>

	

	
		<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;软件负载均衡一般通过两种方式来实现：基于操作系统的软负载实现和基于第三方应用的软负载实现。LVS就是基于Linux操作系统实现的一种软负载，HAProxy就是开源的并且基于第三应用实现的软负载。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;HAProxy相比LVS的使用要简单很多，功能方面也很丰富。当 前，HAProxy支持两种主要的代理模式:”tcp”也即4层（大多用于邮件服务器、内部协议通信服务器等），和7层（HTTP）。在4层模式 下，HAProxy仅在客户端和服务器之间转发双向流量。7层模式下，HAProxy会分析协议，并且能通过允许、拒绝、交换、增加、修改或者删除请求 (request)或者回应(response)里指定内容来控制协议，这种操作要基于特定规则。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;现在用HAProxy主要在于它有以下优点，这里总结下：</p>
<ul>
<li><p>免费开源，稳定性也是非常好，这个可通过我做的一些小项目可以看出来，单Haproxy也跑得不错，稳定性可以与LVS相媲美；</p>
</li>
<li><p>根据官方文档，HAProxy可以跑满10Gbps-New benchmark of HAProxy at 10 Gbps using Myricom’s 10GbE NICs (Myri-10G PCI-Express)，这个作为软件级负载均衡，也是比较惊人的；</p>
</li>
<li><p>HAProxy可以作为MySQL、邮件或其它的非web的负载均衡，我们常用于它作为MySQL(读)负载均衡；</p>
</li>
<li><p>自带强大的监控服务器状态的页面，实际环境中我们结合Nagios进行邮件或短信报警，这个也是我非常喜欢它的原因之一；</p>
</li>
<li><p>HAProxy支持虚拟主机。</p>
</li>
</ul>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在做反向代理服务器的负载均衡时，我们通常会使用nginx的均衡配置。其实，haproxy的负载均衡也是属于这一类的。那么关于这方面的配置过程我们现在来进行一下讲解。首先，对haproxy进行一个简单的介绍，之后就是安装和配置环节了。</p>
<h2 id="HAProxy介绍"><a href="#HAProxy介绍" class="headerlink" title="HAProxy介绍"></a>HAProxy介绍</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;反向代理服务器,支持双机热备支持虚拟主机,但其配置简单,拥有非常不错的服务器健康检查功能,当其代理的后端服务器出现故障, HAProxy会自动将该服务器摘除,故障恢复后再自动将该服务器加入｡新的1.3引入了frontend,backend；frontend根据任意 HTTP请求头内容做规则匹配,然后把请求定向到相关的backend.</p>
<p><a href="http://blog.liuts.com/post/223/" target="_blank" rel="external">http://blog.liuts.com/post/223/</a> （搭建四层负载均衡器）</p>
<h2 id="keepalived简介"><a href="#keepalived简介" class="headerlink" title="keepalived简介　　"></a>keepalived简介　　</h2><p><a href="http://www.keepalived.org" target="_blank" rel="external">http://www.keepalived.org</a></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;keepalived是一个类似于layer3, 4 &amp; 5交换机制的软件，也就是我们平时说的第3层、第4层和第5层交换。Keepalived的作用是检测web服务器的状态，如果有一台web服务器死机，或工作出现故障，Keepalived将检测到，并将有故障的web服务器从系统中剔除，当web服务器工作正常后Keepalived自动将web服务器加入到服务器群中，这些工作全部自动完成，不需要人工干涉，需要人工做的只是修复故障的web服务器。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;类似的HA工具还有heatbeat、drbd等，heatbeat、drbd配置都较为复杂。</p>
<h2 id="keepalived理论工作原理"><a href="#keepalived理论工作原理" class="headerlink" title="keepalived理论工作原理"></a>keepalived理论工作原理</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;keepalived可提供vrrp以及health-check功能，可以只用它提供双机浮动的vip（vrrp虚拟路由功能），这样可以简单实现一个双机热备高可用功能。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;keepalived是一个类似于layer3, 4 &amp; 5交换机制的软件，也就是我们平时说的第3层、第4层和第5层交换。Keepalived的作用是检测web 服务器的状态。 Layer3,4&amp;5工作在IP/TCP协议栈的IP层，TCP层，及应用层,原理分别如下：</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;Layer3：Keepalived使用Layer3的方式工作式时，Keepalived会定期向服务器群中的服务器</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;发送一个ICMP的数据包（既我们平时用的Ping程序）,如果发现某台服务的IP地址没有激活，Keepalived便报告这台服务器失效，并将它从服务器群中剔除，这种情况的典型例子是某台服务器被非法关机。Layer3的方式是以服务器的IP地址是否有效作为服务器工作正常与否的标准。在本文中将采用这种方式。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;Layer4:如果您理解了Layer3的方式，Layer4就容易了。Layer4主要以TCP端口的状态来决定服务器工作正常与否。如web server的服务端口一般是80，如果Keepalived检测到80端口没有启动，则Keepalived将把这台服务器从服务器群中剔除。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;Layer5：Layer5就是工作在具体的应用层了，比Layer3,Layer4要复杂一点，在网络上占用的带宽也要大一些。Keepalived将根据用户的设定检查服务器程序的运行是否正常，如果与用户的设定不相符，则Keepalived将把服务器从服务器群中剔除。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;vip即虚拟ip，是附在主机网卡上的，即对主机网卡进行虚拟，此IP仍然是占用了此网段的某个IP。</p>
<h2 id="keepalived作用"><a href="#keepalived作用" class="headerlink" title="keepalived作用"></a>keepalived作用</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;随着你的网站业务量的增长你网站的服务器压力越来越大？需要负载均衡方案！商业的硬件如F5又太贵，你们又是创业型互联公司如何有效节约成本，节省不必要的浪费？同时实现商业硬件一样的高性能高可用的功能？有什么好的负载均衡可伸张可扩展的方案吗？答案是肯定的！有！我们利用 LVS+Keepalived基于完整开源软件的架构可以为你提供一个负载均衡及高可用的服务器。</p>
<h3 id="LVS-Keepalived-介绍"><a href="#LVS-Keepalived-介绍" class="headerlink" title="LVS+Keepalived 介绍"></a>LVS+Keepalived 介绍</h3><h4 id="LVS"><a href="#LVS" class="headerlink" title="LVS"></a>LVS</h4><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;LVS是Linux Virtual Server的简写，意即Linux虚拟服务器，是一个虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立，是中国国内最早出现的自由软件项目之一.目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR）八种调度算法（rr,wrr,lc,wlc,lblc,lblcr,dh,sh）。</p>
<h4 id="Keepalvied"><a href="#Keepalvied" class="headerlink" title="Keepalvied"></a>Keepalvied</h4><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;Keepalived在这里主要用作RealServer的健康状态检查以及LoadBalance主机和BackUP主机之间failover的实现。keepalived简介　　keepalived是一个类似于layer3, 4 &amp; 5交换机制的软件，也就是我们平时说的第3层、第4层和第5层交换。Keepalived的作用是检测web服务器的状态，如果有一台web服务器死机，或工作出现故障，Keepalived将检测到，并将有故障的web服务器从系统中剔除，当web服务器工作正常后Keepalived自动将web服务器加入到服务器群中，这些工作全部自动完成，不需要人工干涉，需要人工做的只是修复故障的web服务器。</p>
<h2 id="Keepalived介绍"><a href="#Keepalived介绍" class="headerlink" title="Keepalived介绍"></a>Keepalived介绍</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;Keepalived是一个基于VRRP协议来实现的WEB 服务高可用方案，可以利用其来避免单点故障。一个WEB服务至少会有2台服务器运行Keepalived，一台为主服务器（MASTER），一台为备份服务器（BACKUP），但是对外表现为一个虚拟IP，主服务器会发送特定的消息给备份服务器，当备份服务器收不到这个消息的时候，即主服务器宕机的时候，备份服务器就会接管虚拟IP，继续提供服务，从而保证了高可用性。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">+-------------VIP(192.168.0.7)------------------+</div><div class="line">|                         |                      |</div><div class="line">|                         |                      |</div><div class="line">server(MASTER) &lt;----keepalived----&gt; server(BACKUP)</div><div class="line">(192.168.0.1)                        (192.168.0.2)</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;keepalived是VRRP的完美实现，因此在介绍keepalived之前，先介绍一下VRRP的原理。</p>
<h2 id="VRRP协议简介"><a href="#VRRP协议简介" class="headerlink" title="VRRP协议简介"></a>VRRP协议简介</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在现实的网络环境中，两台需要通信的主机大多数情况下并没有直接的物理连接。对于这样的情况，它们之间路由怎样选择？主机如何选定到达目的主机的下一跳路由，这个问题通常的解决方法有二种：</p>
<ul>
<li>在主机上使用动态路由协议(RIP、OSPF等)</li>
<li>在主机上配置静态路由</li>
</ul>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;很明显，在主机上配置路态路由是非常不切实际的，因为管理、维护成本以及是否支持等诸多问题。配置静态路由就变得十分流行，但路由器(或者说默认网关default gateway)却经常成为单点。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;VRRP的目的就是为了解决静态路由单点故障问题。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;VRRP通过一竞选(election)协议来动态的将路由任务交给LAN中虚拟路由器中的某台VRRP路由器。</p>
<h3 id="工作机制"><a href="#工作机制" class="headerlink" title="工作机制"></a>工作机制</h3><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在一个VRRP虚拟路由器中，有多台物理的VRRP路由器，但是这多台的物理的机器并不能同时工作，而是由一台称为MASTER的负责路由工作，其它的都是BACKUP，MASTER并非一成不变，VRRP让每个VRRP路由器参与竞选，最终获胜的就是MASTER。MASTER拥有一些特权，比如 拥有虚拟路由器的IP地址，我们的主机就是用这个IP地址作为静态路由的。拥有特权的MASTER要负责转发发送给网关地址的包和响应ARP请求。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;VRRP通过竞选协议来实现虚拟路由器的功能，所有的协议报文都是通过IP多播(multicast)包(多播地址 224.0.0.18)形式发送的。虚拟路由器由VRID(范围0-255)和一组IP地址组成，对外表现为一个周知的MAC地址。所以，在一个虚拟路由 器中，不管谁是MASTER，对外都是相同的MAC和IP(称之为VIP)。客户端主机并不需要因为MASTER的改变而修改自己的路由配置，对他们来 说，这种主从的切换是透明的。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在一个虚拟路由器中，只有作为MASTER的VRRP路由器会一直发送VRRP广告包(VRRPAdvertisement message)，BACKUP不会抢占MASTER，除非它的优先级(priority)更高。当MASTER不可用时(BACKUP收不到广告包)， 多台BACKUP中优先级最高的这台会被抢占为MASTER。这种抢占是非常快速的（小于1s），以保证服务的连续性。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;由于安全性考虑，VRRP包使用了加密协议进行加密。</p>
<h3 id="vrrp简介"><a href="#vrrp简介" class="headerlink" title="vrrp简介"></a>vrrp简介</h3><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;随着Internet的迅猛发展，基于网络的应用逐渐增多。这就对网络的可靠性提出了越来越高的要求。斥资对所有网络设备进行更新当然是一种很好的可靠性解决方案；但本着保护现有投资的角度考虑，可以采用廉价冗余的思路，在可靠性和经济性方面找到平衡点。 </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;虚拟路由冗余协议就是一种很好的解决方案。在该协议中，对共享多存取访问介质（如以太网）上终端IP设备的默认网关(Default Gateway)进行冗余备份，从而在其中一台路由设备宕机时，备份路由设备及时接管转发工作，向用户提供透明的切换，提高了网络服务质量。  </p>
<h4 id="一、协议概述"><a href="#一、协议概述" class="headerlink" title="一、协议概述"></a>一、协议概述</h4><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在基于TCP/IP协议的网络中，为了保证不直接物理连接的设备之间的通信，必须指定路由。目前常用的指定路由的方法有两种：一种是通过路由协议（比如：内部路由协议RIP和OSPF）动态学习；另一种是静态配置。在每一个终端都运行动态路由协议是不现实的，大多客户端操作系统平台都不支持动态路由协议，即使支持也受到管理开销、收敛度、安全性等许多问题的限制。因此普遍采用对终端IP设备静态路由配置，一般是给终端设备指定一个或者多个默认网关(Default Gateway)。静态路由的方法简化了网络管理的复杂度和减轻了终端设备的通信开销，但是它仍然有一个缺点：如果作为默认网关的路由器损坏，所有使用该网关为下一跳主机的通信必然要中断。即便配置了多个默认网关，如不重新启动终端设备，也不能切换到新的网关。采用虚拟路由冗余协议 (Virtual Router Redundancy Protocol，简称VRRP)可以很好的避免静态指定网关的缺陷。 </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在VRRP协议中，有两组重要的概念：VRRP路由器和虚拟路由器，主控路由器和备份路由器。VRRP路由器是指运行VRRP的路由器，是物理实体，虚拟路由器是指VRRP协议创建的，是逻辑概念。一组VRRP路由器协同工作，共同构成一台虚拟路由器。该虚拟路由器对外表现为一个具有唯一固定IP地址和MAC地址的逻辑路由器。处于同一个VRRP组中的路由器具有两种互斥的角色：主控路由器和备份路由器，一个VRRP组中有且只有一台处于主控角色的路由器，可以有一个或者多个处于备份角色的路由器。VRRP协议使用选择策略从路由器组中选出一台作为主控，负责ARP相应和转发IP数据包，组中的其它路由器作为备份的角色处于待命状态。当由于某种原因主控路由器发生故障时，备份路由器能在几秒钟的时延后升级为主路由器。由于此切换非常迅速而且不用改变IP地址和MAC地址，故对终端使用者系统是透明的。  </p>
<h4 id="二、工作原理"><a href="#二、工作原理" class="headerlink" title="二、工作原理"></a>二、工作原理</h4><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;一个VRRP路由器有唯一的标识：VRID，范围为0—255。该路由器对外表现为唯一的虚拟MAC地址，地址的格式为00-00-5E-00-01-[VRID]。主控路由器负责对ARP请求用该MAC地址做应答。这样，无论如何切换，保证给终端设备的是唯一一致的IP和MAC地址，减少了切换对终端设备的影响。 </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;VRRP控制报文只有一种：VRRP通告(advertisement)。它使用IP多播数据包进行封装，组地址为224.0.0.18，发布范围只限于同一局域网内。这保证了VRID在不同网络中可以重复使用。为了减少网络带宽消耗只有主控路由器才可以周期性的发送VRRP通告报文。备份路由器在连续三个通告间隔内收不到VRRP或收到优先级为0的通告后启动新的一轮VRRP选举。 </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在VRRP路由器组中，按优先级选举主控路由器，VRRP协议中优先级范围是0—255。若VRRP路由器的IP地址和虚拟路由器的接口IP地址相同，则称该虚拟路由器作VRRP组中的IP地址所有者；IP地址所有者自动具有最高优先级：255。优先级0一般用在IP地址所有者主动放弃主控者角色时使用。可配置的优先级范围为1—254。优先级的配置原则可以依据链路的速度和成本、路由器性能和可靠性以及其它管理策略设定。主控路由器的选举中，高优先级的虚拟路由器获胜，因此，如果在VRRP组中有IP地址所有者，则它总是作为主控路由的角色出现。对于相同优先级的候选路由器，按照IP地址大小顺序选举。VRRP还提供了优先级抢占策略，如果配置了该策略，高优先级的备份路由器便会剥夺当前低优先级的主控路由器而成为新的主控路由器。 </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;为了保证VRRP协议的安全性，提供了两种安全认证措施：明文认证和IP头认证。明文认证方式要求：在加入一个VRRP路由器组时，必须同时提供相同的VRID和明文密码。适合于避免在局域网内的配置错误，但不能防止通过网络监听方式获得密码。IP头认证的方式提供了更高的安全性，能够防止报文重放和修改等攻击。 </p>
<h4 id="三、-应用实例"><a href="#三、-应用实例" class="headerlink" title="三、 应用实例"></a>三、 应用实例</h4><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;最典型的VRRP应用：RTA、RTB组成一个VRRP路由器组，假设RTB的处理能力高于RTA，则将RTB配置成IP地址所有者，H1、H2、H3的默认网关设定为RTB。则RTB成为主控路由器，负责ICMP重定向、ARP应答和IP报文的转发；一旦RTB失败，RTA立即启动切换，成为主控，从而保证了对客户透明的安全切换。 </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在VRRP应用中，RTA在线时RTB只是作为后备，不参与转发工作，闲置了路由器RTA和链路L1。通过合理的网络设计，可以到达备份和负载分担双重效果。让RTA、RTB同时属于互为备份的两个VRRP组：在组1中RTA为IP地址所有者；组2中RTB为IP地址所有者。将H1的默认网关设定为RTA；H2、H3的默认网关设定为RTB。这样，既分担了设备负载和网络流量，又提高了网络可靠性。 </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;VRRP协议的工作机理与CISCO公司的HSRP（Hot Standby Routing Protocol）有许多相似之处。但二者主要的区别是在CISCO的HSRP中，需要单独配置一个IP地址作为虚拟路由器对外体现的地址，这个地址不能是组中任何一个成员的接口地址。 </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;使用VRRP协议，不用改造目前的网络结构，最大限度保护了当前投资，只需最少的管理费用，却大大提升了网络性能，具有重大的应用价值。</p>
<h2 id="keepalive的简单应用——管理VIP的飘动"><a href="#keepalive的简单应用——管理VIP的飘动" class="headerlink" title="keepalive的简单应用——管理VIP的飘动"></a>keepalive的简单应用——管理VIP的飘动</h2><p>from:<a href="http://www.cnblogs.com/killkill/archive/2010/12/31/1922360.html" target="_blank" rel="external">http://www.cnblogs.com/killkill/archive/2010/12/31/1922360.html</a></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;VIP的飘动可以为我们解决很多问题，以前我试过使用ifup/ifdown的方式控制网卡的up/down来实现，这种方式有个小问题，就是每次VIP 飘动之后都要等上几十秒才能生效，感觉时间比较长，而且还要配合一些逻辑脚本才能很好地工作，有没有更好的方法呢？当然有，这就是本文的主角—— keepalived。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;安装很简单：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">tar zxvf keepalived-1.1.20.tar.gz</div><div class="line"><span class="built_in">cd</span> keepalived-1.1.20</div><div class="line">./configure --prefix=/</div><div class="line">make</div><div class="line">make install</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;修改一下 /etc/keepalived/keepalived.conf 这个配置文件就可以用了，以下是我的环境，192.168.10.141和192.168.10.142是两个VIP，可以在两台服务器之间飘动：</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/haproxy+keepalived%E5%AE%9E%E7%8E%B0%E9%AB%98%E5%8F%AF%E7%94%A8%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1/01.png?raw=true" alt=""></figure> </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;主机的配置：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div></pre></td><td class="code"><pre><div class="line">global_defs &#123;</div><div class="line">    notification_email &#123;</div><div class="line">        failover@firewall.loc</div><div class="line">    &#125;</div><div class="line">    notification_email_from Alexandre.Cassen@firewall.loc</div><div class="line">    smtp_server 192.168.0.48</div><div class="line">    smtp_connect_timeout 10</div><div class="line">    router_id nginx</div><div class="line">&#125;</div><div class="line">vrrp_instance VI_141 &#123;</div><div class="line">    state BACKUP</div><div class="line">    interface eth0</div><div class="line">    virtual_router_id 141</div><div class="line">    priority 50</div><div class="line">    advert_int 1</div><div class="line">    authentication &#123;</div><div class="line">        auth_type PASS</div><div class="line">        auth_pass 141</div><div class="line">    &#125;</div><div class="line">    virtual_ipaddress &#123;</div><div class="line">        192.168.10.141/26 dev eth0</div><div class="line">    &#125;</div><div class="line">&#125;</div><div class="line"></div><div class="line">vrrp_instance VI_142 &#123;</div><div class="line">    state BACKUP</div><div class="line">    interface eth0</div><div class="line">    virtual_router_id 142</div><div class="line">    priority 100</div><div class="line">    advert_int 1</div><div class="line">    authentication &#123;</div><div class="line">        auth_type PASS</div><div class="line">        auth_pass 142</div><div class="line">    &#125;</div><div class="line">    virtual_ipaddress &#123;</div><div class="line">        192.168.10.142/26 dev eth0</div><div class="line">    &#125;</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;备机的配置：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div></pre></td><td class="code"><pre><div class="line">global_defs &#123;</div><div class="line">    notification_email &#123;</div><div class="line">        failover@firewall.loc</div><div class="line">    &#125;</div><div class="line">    notification_email_from Alexandre.Cassen@firewall.loc</div><div class="line">    smtp_server 10.168.0.48</div><div class="line">    smtp_connect_timeout 10</div><div class="line">    router_id nginx</div><div class="line">&#125;</div><div class="line"></div><div class="line">vrrp_instance VI_141 &#123;</div><div class="line">    state BACKUP</div><div class="line">    interface eth0</div><div class="line">    virtual_router_id 141</div><div class="line">    priority 100</div><div class="line">    advert_int 1</div><div class="line">    authentication &#123;</div><div class="line">    auth_type PASS</div><div class="line">    auth_pass 141</div><div class="line">    &#125;</div><div class="line">    virtual_ipaddress &#123;</div><div class="line">        192.168.10.141/26 dev eth0</div><div class="line">    &#125;</div><div class="line">&#125;</div><div class="line"></div><div class="line">vrrp_instance VI_142 &#123;</div><div class="line">    state BACKUP</div><div class="line">    interface eth0</div><div class="line">    virtual_router_id 142</div><div class="line">    priority 50</div><div class="line">    advert_int 1</div><div class="line">    authentication &#123;</div><div class="line">        auth_type PASS</div><div class="line">        auth_pass 142</div><div class="line">    &#125;</div><div class="line">    virtual_ipaddress &#123;</div><div class="line">        192.168.10.142/26 dev eth0</div><div class="line">    &#125;</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;乍一看，主机和备机的配置文件是一样的，仔细看一下priority的值，使用以下命令即可将keepalived加入linux的服务中：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">chkconfig --add keepalived ;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;通过启、停keepalived这个服务即可观察到VIP的飘动，至于为什么VIP飘动后可以很快地生效，还有待研究。</p>
<h2 id="haproxy-keepalived实现高可用负载均衡"><a href="#haproxy-keepalived实现高可用负载均衡" class="headerlink" title="haproxy+keepalived实现高可用负载均衡"></a>haproxy+keepalived实现高可用负载均衡</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;环境：</p>
<ul>
<li>haproxy keepalived 主：192.168.1.192</li>
<li>haproxy keepalived 备：192.168.1.193</li>
<li>vip：192.168.1.200</li>
<li>web：192.168.1.187:80 192.168.1.187:8000</li>
</ul>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/haproxy+keepalived%E5%AE%9E%E7%8E%B0%E9%AB%98%E5%8F%AF%E7%94%A8%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1/02.jpeg?raw=true" alt=""></figure></p>
<h3 id="一：安装过程，在192-168-1-192上："><a href="#一：安装过程，在192-168-1-192上：" class="headerlink" title="一：安装过程，在192.168.1.192上："></a>一：安装过程，在192.168.1.192上：</h3><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;keepalived的安装:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line">tar -zxvf keepalived-1.1.17.tar.gz</div><div class="line"></div><div class="line">ln -s /usr/src/kernels/2.6.18-128.el5-i686/ /usr/src/linux</div><div class="line"></div><div class="line"><span class="built_in">cd</span> keepalived-1.1.17</div><div class="line"></div><div class="line">./configure --prefix=/ --mandir=/usr/<span class="built_in">local</span>/share/man/ --with-kernel-dir=/usr/src/kernels/2.6.18-128.el5-i686/</div><div class="line"></div><div class="line">make &amp;&amp; make install</div><div class="line"></div><div class="line"><span class="built_in">cd</span> /etc/keepalived/</div><div class="line"></div><div class="line">mv keepalived.conf keepalived.conf.default</div></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line">vi keepalived.conf</div><div class="line"></div><div class="line"></div><div class="line">! Configuration File <span class="keyword">for</span> keepalived</div><div class="line"></div><div class="line">vrrp_script chk_http_port &#123;</div><div class="line">    script <span class="string">"/etc/keepalived/check_haproxy.sh"</span></div><div class="line">    interval 2</div><div class="line">    weight 2</div><div class="line"></div><div class="line">    global_defs &#123;</div><div class="line">        router_id LVS_DEVEL</div><div class="line">    &#125;</div><div class="line">    vrrp_instance VI_1 &#123;</div><div class="line">        state MASTER <span class="comment">#192.168.1.193上改为BACKUP</span></div><div class="line">        interface eth0</div><div class="line">        virtual_router_id 51 </div><div class="line">        priority 150 <span class="comment">#192.168.1.193上改为120</span></div><div class="line">        advert_int 1</div><div class="line">        authentication &#123;</div><div class="line">            auth_type PASS</div><div class="line">            auth_pass 1111</div><div class="line">        &#125;</div><div class="line"></div><div class="line">        track_script &#123;</div><div class="line">            chk_http_port</div><div class="line">        &#125;</div><div class="line"></div><div class="line">        virtual_ipaddress &#123;</div><div class="line">            192.168.1.200 </div><div class="line">        &#125;</div><div class="line">    &#125;</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line">vi /etc/keepalived/check_haproxy.sh</div><div class="line"></div><div class="line"><span class="meta">#!/bin/bash</span></div><div class="line">A=`ps -C haproxy --no-header |wc -l`</div><div class="line"><span class="keyword">if</span> [ <span class="variable">$A</span> -eq 0 ];<span class="keyword">then</span></div><div class="line">    /usr/<span class="built_in">local</span>/haproxy/sbin/haproxy -f /usr/<span class="built_in">local</span>/haproxy/conf/haproxy.cfg</div><div class="line">    sleep 3</div><div class="line">    <span class="keyword">if</span> [ `ps -C haproxy --no-header |wc -l` -eq 0 ];<span class="keyword">then</span></div><div class="line">        /etc/init.d/keepalived stop</div><div class="line">    <span class="keyword">fi</span></div><div class="line"><span class="keyword">fi</span></div></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">chmod 755 /etc/keepalived/check_haproxy.sh</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;haproxy的安装(主备都一样)：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">tar -zxvf haproxy-1.4.9.tar.gz</div><div class="line"><span class="built_in">cd</span> haproxy-1.4.9</div><div class="line">make TARGET=linux26 PREFIX=/usr/<span class="built_in">local</span>/haproxy install</div><div class="line"><span class="built_in">cd</span> /usr/<span class="built_in">local</span>/haproxy/</div><div class="line">mkdir conf logs</div><div class="line"><span class="built_in">cd</span> conf</div></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div></pre></td><td class="code"><pre><div class="line">vi haproxy.cfg</div><div class="line"></div><div class="line">global</div><div class="line"><span class="built_in">log</span> 127.0.0.1 local3 info</div><div class="line">maxconn 4096</div><div class="line">user nobody</div><div class="line">group nobody</div><div class="line">daemon</div><div class="line">nbproc 1</div><div class="line">pidfile /usr/<span class="built_in">local</span>/haproxy/logs/haproxy.pid</div><div class="line"></div><div class="line">defaults</div><div class="line">maxconn 2000</div><div class="line">contimeout 5000</div><div class="line">clitimeout 30000</div><div class="line">srvtimeout 30000</div><div class="line">mode http</div><div class="line"><span class="built_in">log</span> global</div><div class="line"><span class="built_in">log</span> 127.0.0.1 local3 info</div><div class="line">stats uri /admin?stats</div><div class="line">option forwardfor</div><div class="line"></div><div class="line">frontend http_server</div><div class="line"><span class="built_in">bind</span> :80</div><div class="line"><span class="built_in">log</span> global</div><div class="line">default_backend info_cache</div><div class="line">acl <span class="built_in">test</span> hdr_dom(host) -i test.domain.com</div><div class="line">use_backend cache_test <span class="keyword">if</span> <span class="built_in">test</span></div><div class="line"></div><div class="line">backend info_cache</div><div class="line"><span class="comment">#balance roundrobin</span></div><div class="line">balance <span class="built_in">source</span></div><div class="line">option httpchk HEAD /haproxy.txt HTTP/1.1\r\nHost:192.168.1.187</div><div class="line">server inst2 192.168.1.187:80 check inter 5000 fall 3</div><div class="line"></div><div class="line"></div><div class="line">backend cache_test</div><div class="line">balance roundrobin</div><div class="line"><span class="comment">#balance source</span></div><div class="line">option httpchk HEAD /haproxy.txt HTTP/1.1\r\nHost:test.domain.com</div><div class="line">server inst1 192.168.1.187:8000 check inter 5000 fall 3</div></pre></td></tr></table></figure>
<h3 id="二：再两台机器上都分别启动："><a href="#二：再两台机器上都分别启动：" class="headerlink" title="二：再两台机器上都分别启动："></a>二：再两台机器上都分别启动：</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">/etc/init.d/keepalived start <span class="comment">#这条命令会自动把haproxy启动</span></div></pre></td></tr></table></figure>
<h3 id="三：测试："><a href="#三：测试：" class="headerlink" title="三：测试："></a>三：测试：</h3><ol>
<li>再两台机器上分别执行ip add</li>
</ol>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;主:</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast qlen 1000</div><div class="line">link/ether 00:0c:29:98:<span class="built_in">cd</span>:c0 brd ff:ff:ff:ff:ff:ff</div><div class="line">inet 192.168.1.192/24 brd 192.168.1.255 scope global eth0</div><div class="line">inet 192.168.1.200/32 scope global eth0</div><div class="line">inet6 fe80::20c:29ff:fe98:cdc0/64 scope link</div><div class="line">valid_lft forever preferred_lft forever</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;备: </p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast qlen 1000</div><div class="line">link/ether 00:0c:29:a6:0c:7e brd ff:ff:ff:ff:ff:ff</div><div class="line">inet 192.168.1.193/24 brd 255.255.255.254 scope global eth0</div><div class="line">inet6 fe80::20c:29ff:fea6:c7e/64 scope link</div><div class="line">valid_lft forever preferred_lft forever</div></pre></td></tr></table></figure>
<ol>
<li><p>停掉主上的haproxy，3秒后keepalived会自动将其再次启动</p>
</li>
<li><p>停掉主的keepalived，备机马上接管服务</p>
</li>
</ol>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;备: </p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast qlen 1000</div><div class="line">link/ether 00:0c:29:a6:0c:7e brd ff:ff:ff:ff:ff:ff</div><div class="line">inet 192.168.1.193/24 brd 255.255.255.254 scope global eth0</div><div class="line">inet 192.168.1.200/32 scope global eth0</div><div class="line">inet6 fe80::20c:29ff:fea6:c7e/64 scope link</div><div class="line">valid_lft forever preferred_lft forever</div></pre></td></tr></table></figure>
<ol>
<li>更改hosts</li>
</ol>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">192.168.1.200 test.com</div><div class="line">192.168.1.200 test.domain.com</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;通过IE测试，可以发现</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;test.com的请求发向了192.168.1.187:80</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;test.domain.com的请求发向了192.168.1.187:8000</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/haproxy+keepalived%E5%AE%9E%E7%8E%B0%E9%AB%98%E5%8F%AF%E7%94%A8%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1/03.jpeg?raw=true" alt=""></figure></p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/haproxy+keepalived%E5%AE%9E%E7%8E%B0%E9%AB%98%E5%8F%AF%E7%94%A8%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1/04.jpeg?raw=true" alt=""></figure></p>

	

	

</article>




	<article>
	
		<h1><a href="/2017/10/12/LVS/2. LVS 的 nat 模式/">LVS 的 nat 模式</a></h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/LVS/">LVS</a>
			</span>
		
	</div>

	

	
		<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84nat%E6%A8%A1%E5%BC%8F/01.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;<strong>类似于iptables的DNAT，但支持多目标转发。客户端和realserver不能再同一个网段，不然直接响应，不走网关</strong></p>
<h2 id="1-模块讲解"><a href="#1-模块讲解" class="headerlink" title="1.模块讲解"></a>1.模块讲解</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;客户端发起请求</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;nat模式中，vip机器进行ip的转发，只改变目的ip</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;res1和res2需要提供web服务(nginx或者httpd都可以)</p>
<h2 id="2-请求流程图"><a href="#2-请求流程图" class="headerlink" title="2.请求流程图"></a>2.请求流程图</h2><ol>
<li>客户端发起请求到vip机器上，vip服务器根据lvs的算法，转发给realserver服务器（改变目的ip为res1或者res2），并记录连接信息，只改变目的ip，源ip不变。</li>
<li>real-server收到request请求包之后，发现目的ip是自己的ip，处理请求，然后走网关，经过vip</li>
<li>vip收到reply包后，修改reply包的源ip地址为vip，发给客户端</li>
<li>从客户端来的属于本地连接的包，查hash表，然后转发给real-server</li>
<li>当client发送完毕以后，此次连接结束或者连接超时，lvs自动从hast表中删除此条记录；</li>
</ol>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;数据包流程如下：</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;1.客户端发起请求：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="built_in">source</span>      	192.168.147.1			dest    192.168.147.150:80</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在vip服务器上，eth1网卡上抓包如下：</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84nat%E6%A8%A1%E5%BC%8F/02.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;2.lvs(vip)处理请求：</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;vip服务器上抓包，在eth1网卡：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="built_in">source</span>		192.168.147.1:59334		dst	192.168.1.112/111:80</div></pre></td></tr></table></figure>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84nat%E6%A8%A1%E5%BC%8F/03.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;3.realserver处理完请求以后，网关是eth0；在vip的eth0上抓包如下</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="built_in">source</span>             192.168.1.112              dst    192.168.147.1</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;4.包经过eth0返回vip服务器，然后经过eth1返回给客户端，源ip改成vip，目的ip不变，</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="built_in">source</span>		192.168.147.150           dst        192.168.147.1</div></pre></td></tr></table></figure>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84nat%E6%A8%A1%E5%BC%8F/04.jpeg?raw=true" alt=""></figure></p>
<h2 id="lvs-nat模式部署过程："><a href="#lvs-nat模式部署过程：" class="headerlink" title="lvs_nat模式部署过程："></a>lvs_nat模式部署过程：</h2><ol>
<li>（确保所有服务器的iptables   selinux都是关闭状态）</li>
<li>两个realserver上安装nginx或者http提供web服务</li>
<li>vip上安装ipvsadm </li>
<li>在vip服务器上新建lvs_nat.sh脚本，然后执行</li>
</ol>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#! /bin/bash</span></div><div class="line"><span class="comment"># director 服务器上开启路由转发功能: </span></div><div class="line"><span class="built_in">echo</span> 1 &gt; /proc/sys/net/ipv4/ip_forward </div><div class="line"><span class="comment"># 关闭icmp的重定向</span></div><div class="line"><span class="built_in">echo</span> 0 &gt; /proc/sys/net/ipv4/conf/all/send_redirects</div><div class="line"><span class="built_in">echo</span> 0 &gt; /proc/sys/net/ipv4/conf/default/send_redirects</div><div class="line"><span class="built_in">echo</span> 0 &gt; /proc/sys/net/ipv4/conf/eth0/send_redirects</div><div class="line"><span class="built_in">echo</span> 0 &gt; /proc/sys/net/ipv4/conf/eth1/send_redirects</div><div class="line"></div><div class="line">iptables -F</div><div class="line">iptables -t nat -F</div><div class="line">iptables -t nat -X</div><div class="line"><span class="comment">#iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -j MASQUERADE</span></div><div class="line"></div><div class="line">ipvsadm -C</div><div class="line">ipvsadm -A -t 192.168.147.150:80 -s rr -p 300</div><div class="line"><span class="comment">#-p 300   长链接300秒</span></div><div class="line">ipvsadm -a -t 192.168.147.150:80 -r 192.168.1.111:80 -m -w 1</div><div class="line">ipvsadm -a -t 192.168.147.150:80 -r 192.168.1.112:80 -m -w 1</div></pre></td></tr></table></figure>
<ol>
<li>realserver的网关是eth0，及vip的内网ip</li>
</ol>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;<strong>lvs——nat模式的缺点：当流量比较高时，因为所有的流量进入和出去都要进过lvs，所以瓶颈就在lvs上，而dr模式就比这个效果好很多</strong></p>
<h2 id="待解决问题"><a href="#待解决问题" class="headerlink" title="待解决问题"></a>待解决问题</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;当访问res2的时候，在111上可以抓到112:80到147.1上的包，如下：</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84nat%E6%A8%A1%E5%BC%8F/05.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;当访问res1时，在111上抓不到112:80到147.1上的包</p>

	

	

</article>




	<article>
	
		<h1><a href="/2017/10/12/LVS/19. LVS结合keepalived配置/">LVS结合keepalived配置</a></h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/LVS/">LVS</a> <a class="article__tag-link" href="/tags/keepalived/">keepalived</a>
			</span>
		
	</div>

	

	
		<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;因为 NAT、DR 已经配置过一些操作，而使用 keepalived 操作和以前的操作是有些冲突的，所以先做一些处理。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在 dr 上执行 ：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">[root@dir ~]<span class="comment"># ipvsadm -C</span></div><div class="line">[root@dir ~]<span class="comment"># ifconfig eth0:0 down</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;为什么要引入 keepalived ，之前的 lvs 虽然已经配置成功也实现了负载均衡，但是测试的时候发现，当某台 real server 把 nginx 停掉，那么 director 照样会把请求转发过去，这样就造成了某些请求不正常。所以需要有一种机制用来检测 real server 的状态，也就是 keepalived 。它的作用除了可以检测 RS 状态外，还可以检测备用 director 的状态，也就是说 keepalived 可以实现 HA 集群的功能，当然也需要一台备用 director 服务器。备用 director 也需要安装一下 keepalived 软件。两台 director 上都执行命令：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">yum install -y keepalived</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;主 director 上安装好后，编辑配置文件</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@dir ~]<span class="comment"># vim /etc/keepalived/keepalived.conf</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;加入或更改如下配置</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div></pre></td><td class="code"><pre><div class="line">vrrp_instance VI_1 &#123;</div><div class="line">    state MASTER <span class="comment">#备用服务器上为 BACKUP</span></div><div class="line">    interface eth0</div><div class="line">    virtual_router_id 51</div><div class="line">    priority 100 <span class="comment">#备用服务器上为90</span></div><div class="line">    advert_int 1</div><div class="line">    authentication &#123;</div><div class="line">        auth_type PASS</div><div class="line">        auth_pass 1111</div><div class="line">    &#125;</div><div class="line">    virtual_ipaddress &#123;</div><div class="line">        192.168.0.64</div><div class="line">    &#125;</div><div class="line">&#125;</div><div class="line">virtual_server 192.168.0.64 80 &#123;</div><div class="line">    delay_loop 6 <span class="comment">#(每隔10秒查询realserver状态)</span></div><div class="line">    lb_algo wlc <span class="comment">#(lvs 算法)</span></div><div class="line">    lb_kind DR <span class="comment">#(Direct Route)</span></div><div class="line">    persistence_timeout 60 <span class="comment">#(同一IP的连接60秒内被分配到同一台realserver)</span></div><div class="line">    protocol TCP <span class="comment">#(用TCP协议检查realserver状态)</span></div><div class="line"></div><div class="line">    real_server 192.168.0.66 80 &#123;</div><div class="line">        weight 100 <span class="comment">#(权重)</span></div><div class="line">        TCP_CHECK &#123;</div><div class="line">        connect_timeout 10 <span class="comment">#(10秒无响应超时)</span></div><div class="line">        nb_get_retry 3</div><div class="line">        delay_before_retry 3</div><div class="line">        connect_port 80</div><div class="line">        &#125;</div><div class="line">    &#125;</div><div class="line">    real_server 192.168.0.65 80 &#123;</div><div class="line">        weight 100</div><div class="line">        TCP_CHECK &#123;</div><div class="line">        connect_timeout 10</div><div class="line">        nb_get_retry 3</div><div class="line">        delay_before_retry 3</div><div class="line">        connect_port 80</div><div class="line">        &#125;</div><div class="line">    &#125;</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;以上为主 director 的配置文件，从 director 的配置文件只需要修改如下内容</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">state MASTER -&gt; state BACKUP</div><div class="line">priority 100 -&gt; priority 90</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;更改后从配置为</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div></pre></td><td class="code"><pre><div class="line">vrrp_instance VI_1 &#123;</div><div class="line">    state BACKUP <span class="comment">#主服务器上为 MASTER</span></div><div class="line">    interface eth0</div><div class="line">    virtual_router_id 51</div><div class="line">    priority 90 <span class="comment">#主服务器上为100</span></div><div class="line">    advert_int 1</div><div class="line">    authentication &#123;</div><div class="line">        auth_type PASS</div><div class="line">        auth_pass 1111</div><div class="line">    &#125;</div><div class="line">    virtual_ipaddress &#123;</div><div class="line">        192.168.0.64</div><div class="line">    &#125;</div><div class="line">&#125;</div><div class="line">virtual_server 192.168.0.64 80 &#123;</div><div class="line">    delay_loop 6 <span class="comment">#(每隔10秒查询realserver状态)</span></div><div class="line">    lb_algo wlc <span class="comment">#(lvs 算法)</span></div><div class="line">    lb_kind DR <span class="comment">#(Direct Route)</span></div><div class="line">    persistence_timeout 60 <span class="comment">#(同一IP的连接60秒内被分配到同一台realserver)</span></div><div class="line">    protocol TCP <span class="comment">#(用TCP协议检查realserver状态)</span></div><div class="line"></div><div class="line">    real_server 192.168.0.66 80 &#123;</div><div class="line">        weight 100 <span class="comment">#(权重)</span></div><div class="line">        TCP_CHECK &#123;</div><div class="line">        connect_timeout 10 <span class="comment">#(10秒无响应超时)</span></div><div class="line">        nb_get_retry 3</div><div class="line">        delay_before_retry 3</div><div class="line">        connect_port 80</div><div class="line">        &#125;</div><div class="line">    &#125;</div><div class="line">    real_server 192.168.0.65 80 &#123;</div><div class="line">        weight 100</div><div class="line">        TCP_CHECK &#123;</div><div class="line">        connect_timeout 10</div><div class="line">        nb_get_retry 3</div><div class="line">        delay_before_retry 3</div><div class="line">        connect_port 80</div><div class="line">        &#125;</div><div class="line">    &#125;</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;配置完成 keepalived 后，需要开启端口转发（主从都要做）</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="built_in">echo</span> 1 &gt; /proc/sys/net/ipv4/ip_forward</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;然后，两个 rs 上执行 /usr/local/sbin/lvs_dr_rs.sh 脚本</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">sh /usr/<span class="built_in">local</span>/sbin/lvs_dr_rs.sh</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;最后，两个 director 上启动 keepalived 服务（先主后从）</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">/etc/init.d/keepalived start</div></pre></td></tr></table></figure>

	

	

</article>




	<article>
	
		<h1><a href="/2017/10/12/LVS/18. 只有一个公网IP也可以使用LVS的DR模式！(外带php session粘滞问题解决）/">只有一个公网IP也可以使用LVS的DR模式！(外带php session粘滞问题解决）</a></h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/LVS/">LVS</a>
			</span>
		
	</div>

	

	
		<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/%E5%8F%AA%E6%9C%89%E4%B8%80%E4%B8%AA%E5%85%AC%E7%BD%91IP%E4%B9%9F%E5%8F%AF%E4%BB%A5%E4%BD%BF%E7%94%A8LVS%E7%9A%84DR%E6%A8%A1%E5%BC%8F/01.jpeg?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;单个公网ip怎么才能使用LVS的DR模式？倒不是因为没有公网IP，而是由于安全性的考虑不希望服务器都暴漏在外，又不想因为这个小项目买防火墙，所以就提了这个要求。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;怎么样才能实现呢？一个公网IP也可以做DR啊，前面加个路由器就可以了</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;具体结构就想上面那个图那样</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;原理就是让 路由器把所有的80端口请求都分给VIP，分发器再分给每个web服务器，而web服务器处理完请求后跟客户连接就不走分发器了，直接通过路由器去外网了，这样就实现了只用一个公网IP也能用DR模式，呵呵 具体配置如下</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;先从内网找了三台服务器分别是：</p>
<ul>
<li>192.168.1.166 web1</li>
<li>192.168.1.167 web2</li>
<li>192.168.1.160 分发器</li>
<li>192.168.1.169 VIP</li>
</ul>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;192.168.1.1   路由器内网ip(网关) 路由器是随便找的一台tplink adal路由器，凑合着测试用的</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;211.83.113.119 路由器的WAN口IP </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;先安装ipvsadm 直接yum install ipvsadm就行了</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;用的是keepalived，配置文件贴上来，以下是分发器上的设置</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div></pre></td><td class="code"><pre><div class="line">global_defs &#123;</div><div class="line">   notification_email &#123;</div><div class="line">        ufo@xman.com</div><div class="line">   &#125;</div><div class="line"></div><div class="line">notification_email_from Alexandre.Cassen@firewall.loc</div><div class="line">   smtp_server smtp.qq.com</div><div class="line">   smtp_connect_timeout 30</div><div class="line">   router_id LVS_DEVEL</div><div class="line">&#125;</div><div class="line"></div><div class="line">vrrp_sync_group VG1 &#123;</div><div class="line">        group&#123;</div><div class="line">        VI_1</div><div class="line">                &#125;</div><div class="line">                    &#125;</div><div class="line">vrrp_instance VI_1 &#123;</div><div class="line">        state MASTER</div><div class="line">        interface eth0</div><div class="line">        virtual_router_id 51</div><div class="line">        priority 100</div><div class="line">        advert_int 1</div><div class="line"></div><div class="line">        authentication &#123;</div><div class="line">        auth_type PASS</div><div class="line">        auth_pass 33210</div><div class="line">                        &#125;</div><div class="line"></div><div class="line">        virtual_ipaddress &#123;</div><div class="line">        192.168.1.169</div><div class="line">                          &#125;</div><div class="line"></div><div class="line">        virtual_server 192.168.1.169 80 &#123;</div><div class="line">        delay_loop 6</div><div class="line">        lb_algo rr</div><div class="line">        lb_kind DR</div><div class="line">        protocol TCP</div><div class="line"></div><div class="line">                real_server 192.168.1.166 80 &#123;</div><div class="line">                weight 1</div><div class="line">                inhibit_on_failure</div><div class="line">                TCP_CHECK &#123;</div><div class="line">                connect_timeout 5</div><div class="line">                nb_get_retry 3</div><div class="line">                delay_before_retry 3</div><div class="line">                connect_port 80</div><div class="line">                          &#125;</div><div class="line">                                              &#125;</div><div class="line">                real_server 192.168.1.167 80 &#123;</div><div class="line">                weight 1</div><div class="line">                inhibit_on_failure</div><div class="line">                TCP_CHECK &#123;</div><div class="line">                connect_timeout 5</div><div class="line">                nb_get_retry 3</div><div class="line">                delay_before_retry 3</div><div class="line">                connect_port 80</div><div class="line">                          &#125;</div><div class="line">                                              &#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;配置文件写完了，然后就是</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">mkdir /etc/keepalived  <span class="comment">#系统默认会到这里去找配置文件</span></div><div class="line">cp /usr/<span class="built_in">local</span>/keepalive/etc/keepalived/keepalived.conf /etc/keepalived/</div><div class="line">cp /usr/<span class="built_in">local</span>/keepalive/etc/rc.d/init.d/keepalived /etc/init.d/</div><div class="line">cp /usr/<span class="built_in">local</span>/keepalive/etc/sysconfig/keepalived /etc/sysconfig/</div><div class="line">cp /usr/<span class="built_in">local</span>/keepalive/sbin/keepalived /bin/ <span class="comment">#将可执行程序放入sbin 或者 bin目录里</span></div></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">vim /etc/sysctl.conf</div><div class="line"></div><div class="line">net.ipv4.ip_forward = 1</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;保存退出 后执行</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">sysctl -p</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;route add defaule gw 192.168.1.1 把路由内网地址添加为默认网关</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;<strong>web服务器设置</strong></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;两台web服务器也要修改 /etc/sysctl.conf 修改内容如下</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">vim /etc/sysctl.conf</div><div class="line"><span class="comment"># LVS</span></div><div class="line">net.ipv4.conf.all.arp_ignore = 1</div><div class="line">net.ipv4.conf.all.arp_announce = 2</div><div class="line">net.ipv4.conf.lo.arp_ignore = 1</div><div class="line">net.ipv4.conf.lo.arp_announce = 2</div><div class="line">sysctl -p</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;之后还要增加vip</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;ifconfig lo:1 192.168.1.169 netmask 255.255.255.255 别忘了加到rc.local里面</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;route add defaule gw 192.168.1.1 把路由内网地址添加为默认网关</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;<strong>路由器设置</strong></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;路由器的设置没什么好说的，除了上网设置以外还要做一个端口映射，就是把80端口映射到 vip上也就是192.168.1.169</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;现在启动keepalived吧</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">/etc/init.d/keepalived start</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;开始的时候比较慢，大概1分钟后系统日志里面出现下面这条记录就OK了</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">avahi-daemon[3012]: Registering new address record <span class="keyword">for</span> 192.168.1.169 on eth0</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;访问一下 <a href="http://211.83.113.119" target="_blank" rel="external">http://211.83.113.119</a> 成功了 </p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">ipvsadm -L -n</div><div class="line">IP Virtual Server version 1.2.1 (size=4096)</div><div class="line">Prot LocalAddress:Port Scheduler Flags</div><div class="line">  -&gt; RemoteAddress:Port           Forward Weight ActiveConn InActConn</div><div class="line">TCP  192.168.1.169:80 rr</div><div class="line">  -&gt; 192.168.1.166:80             Route   1      5          6         </div><div class="line">  -&gt; 192.168.1.167:80             Route   1      3          9</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;后来遇到了一个问题，由于这套应用处在一个大网站的后台，所以大部分的请求都来自同一个IP地址，而有一部分程序需要给每个连接做session粘滞，</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;这样就不能用lvs 的-p参数来设置ip粘滞时间，如果用lvs的粘滞时间的话大部分的请求都将分给同一台web服务器(注意：这里是session粘滞而不是IP粘滞)，</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;lvs可做不到这点，怎么办呢？</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;解决办法是 将session共享，共享到什么地方就有很多选择了</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;这里是把所有web服务器的php session都给memcached ，这样不管分发器把 ip连接分给哪个web服务器都不会有问题了，配置方法很简单，就在php的配置文件内</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;增加一条语句就可以了，不过前提需要装好memcache模块</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">[Session]</div><div class="line">; Handler used to store/retrieve data.</div><div class="line">session.save_handler = memcache</div><div class="line">session.save_path = <span class="string">"tcp://192.168.1.161:11213"</span></div></pre></td></tr></table></figure>

	

	

</article>




	<article>
	
		<h1><a href="/2017/10/12/LVS/17. LVS的NAT设置/">LVS的NAT设置</a></h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/LVS/">LVS</a>
			</span>
		
	</div>

	

	
		<h2 id="1-环境说明"><a href="#1-环境说明" class="headerlink" title="1.环境说明"></a>1.环境说明</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;三台服务器一台作为 director ，两台作为 real server ，Diretcor 有一个外网 ip （192.168.119.110）和一个内网 ip（192.168.0.67），两个 real server 上只有内网 ip （192.168.0.66）和（192.168.0.65）并且需要把两个 real server 的内网网关设置为 diretcor 的内网 ip （192.168.0.67）</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;更改 hostname </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;dir 192.168.0.67</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;rs1 192.168.0.66</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;rs2 192.168.0.65</p>
<h2 id="2-安装和配置"><a href="#2-安装和配置" class="headerlink" title="2.安装和配置"></a>2.安装和配置</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;两个 real server 上都安装 nginx 服务</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">yum install -y nginx</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;director 上安装 ipvsadm</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@dir ~]<span class="comment"># yum install -y ipvsadm</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;Director 上编辑 nat 实现脚本</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@dir ~]<span class="comment"># vim /usr/local/sbin/lvs_nat.sh</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;写入一下内容</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#! /bin/bash</span></div><div class="line"><span class="comment"># director 服务器上开启路由转发功能: </span></div><div class="line"><span class="built_in">echo</span> 1 &gt; /proc/sys/net/ipv4/ip_forward </div><div class="line"><span class="comment"># 关闭icmp的重定向</span></div><div class="line"><span class="built_in">echo</span> 0 &gt; /proc/sys/net/ipv4/conf/all/send_redirects</div><div class="line"><span class="built_in">echo</span> 0 &gt; /proc/sys/net/ipv4/conf/default/send_redirects</div><div class="line"><span class="built_in">echo</span> 0 &gt; /proc/sys/net/ipv4/conf/eth0/send_redirects</div><div class="line"><span class="built_in">echo</span> 0 &gt; /proc/sys/net/ipv4/conf/eth1/send_redirects</div><div class="line"></div><div class="line"><span class="comment"># director 设置nat防火墙</span></div><div class="line">iptables -t nat -F</div><div class="line">iptables -t nat -X</div><div class="line">iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -j MASQUERADE</div><div class="line"><span class="comment"># director设置ipvsadm</span></div><div class="line">IPVSADM=<span class="string">'/sbin/ipvsadm'</span></div><div class="line"><span class="variable">$IPVSADM</span> -C</div><div class="line"><span class="variable">$IPVSADM</span> -A -t 192.168.119.110:80 -s lc -p 300</div><div class="line"><span class="variable">$IPVSADM</span> -a -t 192.168.119.110:80 -r 192.168.0.66:80 -m -w 1</div><div class="line"><span class="variable">$IPVSADM</span> -a -t 192.168.119.110:80 -r 192.168.0.65:80 -m -w 1</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;保存后，在 Director 上直接运行这个脚本就可以完成 lvs/nat 的配置了。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@dir ~]<span class="comment"># sh /usr/local/sbin/lvs_nat.sh</span></div></pre></td></tr></table></figure>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84NAT%E8%AE%BE%E7%BD%AE/01.png?raw=true" alt=""></figure></p>
<h2 id="3-测试"><a href="#3-测试" class="headerlink" title="3.测试"></a>3.测试</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;通过浏览器测试两台机器上的 web 内容 <a href="http://192.168.119.110" target="_blank" rel="external">http://192.168.119.110</a> 。为了区分开，可以把 nginx 的默认页修改一下</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在 rs1 上执行</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@rs1 ~]<span class="comment"># echo "rs1rs1" &gt; /usr/share/nginx/html/index.html</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在 rs2 上执行</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@rs2 ~]<span class="comment"># echo "rs2rs2" &gt; /usr/share/nginx/html/index.html</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;注意，切记一定要在两台 rs </p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84NAT%E8%AE%BE%E7%BD%AE/02.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;刷新一下</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84NAT%E8%AE%BE%E7%BD%AE/03.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;或者使用 curl</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84NAT%E8%AE%BE%E7%BD%AE/04.png?raw=true" alt=""></figure></p>

	

	

</article>




	<article>
	
		<h1><a href="/2017/10/12/LVS/16. LVS的DR设置/">LVS的DR设置</a></h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/LVS/">LVS</a>
			</span>
		
	</div>

	

	
		<h2 id="1-环境说明"><a href="#1-环境说明" class="headerlink" title="1.环境说明"></a>1.环境说明</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;三台机器</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;director  （eth0： 192.168.0.67， vip eth0:0： 192.168.0.64）</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;real server1 （eth0 rip：192.168.0.66 ，vip lo:0：192.168.0.64）</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;real server2 （eth0 rip：192.168.0.65 ，vip lo:0：192.168.0.64）</p>
<h2 id="2-编写脚本"><a href="#2-编写脚本" class="headerlink" title="2.编写脚本"></a>2.编写脚本</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;director 上执行</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@dir ~]<span class="comment"># vim /usr/local/sbin/lvs_dr.sh</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;增加内容</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#! /bin/bash</span></div><div class="line"><span class="built_in">echo</span> 1 &gt; /proc/sys/net/ipv4/ip_forward</div><div class="line">ipv=/sbin/ipvsadm</div><div class="line">vip=192.168.0.64</div><div class="line">rs1=192.168.0.66</div><div class="line">rs2=192.168.0.65</div><div class="line">ifconfig eth0:0 <span class="variable">$vip</span> broadcast <span class="variable">$vip</span> netmask 255.255.255.255 up</div><div class="line">route add -host <span class="variable">$vip</span> dev eth0:0</div><div class="line"><span class="variable">$ipv</span> -C</div><div class="line"><span class="variable">$ipv</span> -A -t <span class="variable">$vip</span>:80 -s rr </div><div class="line"><span class="variable">$ipv</span> -a -t <span class="variable">$vip</span>:80 -r <span class="variable">$rs1</span>:80 -g -w 1</div><div class="line"><span class="variable">$ipv</span> -a -t <span class="variable">$vip</span>:80 -r <span class="variable">$rs2</span>:80 -g -w 1</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;保存后在两台 rs 上执行</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">vim /usr/<span class="built_in">local</span>/sbin/lvs_dr_rs.sh</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;增加内容</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#! /bin/bash</span></div><div class="line">vip=192.168.0.64</div><div class="line">ifconfig lo:0 <span class="variable">$vip</span> broadcast <span class="variable">$vip</span> netmask 255.255.255.255 up </div><div class="line">route add -host <span class="variable">$vip</span> lo:0</div><div class="line"><span class="built_in">echo</span> <span class="string">"1"</span> &gt;/proc/sys/net/ipv4/conf/lo/arp_ignore</div><div class="line"><span class="built_in">echo</span> <span class="string">"2"</span> &gt;/proc/sys/net/ipv4/conf/lo/arp_announce</div><div class="line"><span class="built_in">echo</span> <span class="string">"1"</span> &gt;/proc/sys/net/ipv4/conf/all/arp_ignore</div><div class="line"><span class="built_in">echo</span> <span class="string">"2"</span> &gt;/proc/sys/net/ipv4/conf/all/arp_announce</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;关于 arp_ignore 和 arp_announce 参考：<a href="https://hcldirgit.github.io/2017/08/07/15.%20LVS%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1%E4%B8%ADarp_ignore%E5%92%8Carp_annonuce%E5%8F%82%E6%95%B0%E9%85%8D%E7%BD%AE%E7%9A%84%E5%90%AB%E4%B9%89/" target="_blank" rel="external">LVS负载均衡中arp_ignore和arp_annonuce参数配置的含义</a></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;然后 driector 上执行</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@dir ~]<span class="comment"># sh /usr/local/sbin/lvs_dr.sh</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;两台 rs 上执行</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">sh /usr/<span class="built_in">local</span>/sbin/lvs_dr_rs.sh</div></pre></td></tr></table></figure>
<h2 id="3-测试"><a href="#3-测试" class="headerlink" title="3.测试"></a>3.测试</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;windows 下浏览器测试访问 <a href="http://192.168.0.64" target="_blank" rel="external">http://192.168.0.64</a></p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84DR%E8%AE%BE%E7%BD%AE/03.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;更换浏览器</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84DR%E8%AE%BE%E7%BD%AE/01.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;或者换台计算机用 curl</p>
<p> <figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/LVS%E7%9A%84DR%E8%AE%BE%E7%BD%AE/02.png?raw=true" alt=""></figure></p>

	

	

</article>




	<article>
	
		<h1><a href="/2017/10/12/LVS/15. LVS负载均衡中arp_ignore和arp_annonuce参数配置的含义/">LVS负载均衡中arp_ignore和arp_annonuce参数配置的含义</a></h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/LVS/">LVS</a>
			</span>
		
	</div>

	

	
		<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;先简单的介绍下关于LVS负载均衡</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;LVS（Linux  Virtual Server）Linux服务器集群系统</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;针对高可伸缩，高可用服务的需求，给予IP层和内容请求分发的负载均衡调度解决方法，并在Linux的内核中实现，将一组服务器构成一个实现可伸缩，高可用网络服务的虚拟服务器</p>
<h2 id="负载均衡"><a href="#负载均衡" class="headerlink" title="负载均衡"></a>负载均衡</h2><ol>
<li>大量的兵法访问或数据流量分担到多态节点设备分别处理，减少用户的等待时间</li>
<li>单个重负载的运算分担到多态节点设备上做并行处理，每个节点设备处理结束后，将结果汇总，返回给用户</li>
</ol>
<h2 id="负载调度器"><a href="#负载调度器" class="headerlink" title="负载调度器"></a>负载调度器</h2><p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;一组服务器通过高速的局域网或者地理分布的广域网相互相连，在他们的前端有一个负载均衡调度器（Load Balancer），负载均衡调度器能无缝的将网络请求调度到真实的服务器上，从而使得服务器集群的结构对用户是透明的，用户通过访问集群系统提供的网络服务，就像访问一台高性能，高可用的服务器。</p>
<h2 id="IP负载均衡技术（三种）"><a href="#IP负载均衡技术（三种）" class="headerlink" title="IP负载均衡技术（三种）"></a>IP负载均衡技术（三种）</h2><ol>
<li><p>VS/NAT（网络地址转换）<br>通过网络地址转换，调度器重写请求报文的目标地址，根据预设的调度算法，将请求分发给后端的真实服务器，真实服务器的响应报文通过调度器时，报文的源地址被重写，再返回到客户端，完成整个调度的过程</p>
</li>
<li><p>VS/TUN（IP隧道模式）<br>调度器将请求的报文通过IP隧道转发至真实服务器，而真实的服务器直接将结果返回给用户，调度器只处理请求报文，由于一般网路服务的应答大于请求，采用IP隧道模式，集群系统的最大吞吐量可以提高10倍。</p>
</li>
<li><p>VS/DR（直接路由）<br>通过改写请求报文的MAC地址，将请求发送到真是服务器，真实服务器将响应直接返回给用户，之际额路由模式可以极大的提高集群系统的伸缩性，这种方法没有IP隧道的开销，集群中真实的服务器也没有必要必须支持IP隧道协议，只是需要调度器与真实服务器有一块网卡连在同一物理网段上。</p>
</li>
</ol>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;其中在这三种IP负载均衡的技术中，DR和TUN模式都需要在真实服务器上对arp_ignore和arp_announce参数进行配置，主要是实现禁止响应对VIP的ARP请求。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在lvs环境中，需要设定以下的参数 </p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="built_in">echo</span> <span class="string">"1"</span> &gt; /proc/sys/net/ipv4/conf/all/arp_ignore</div><div class="line"><span class="built_in">echo</span> <span class="string">"1"</span> &gt; /proc/sys/net/ipv4/conf/lo/arp_ignore</div><div class="line"><span class="built_in">echo</span> <span class="string">"2"</span> &gt; /proc/sys/net/ipv4/conf/lo/arp_announce</div><div class="line"><span class="built_in">echo</span> <span class="string">"2"</span> &gt; /proc/sys/net/ipv4/conf/all/arp_announce</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;先来看看关于arp_ignore和arp_announce的有关介绍</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;有关<strong>arp_ignore</strong>的相关介绍：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">arp_ignore - INTEGER Define different modes <span class="keyword">for</span> sending replies <span class="keyword">in</span> response to received ARP requests that resolve <span class="built_in">local</span> target IP addresses: 0 - (default): reply <span class="keyword">for</span> any <span class="built_in">local</span> target IP address, configured on any interface 1 - reply only <span class="keyword">if</span> the target IP address is <span class="built_in">local</span> address configured on the incoming interface 2 - reply only <span class="keyword">if</span> the target IP address is <span class="built_in">local</span> address configured on the incoming interface and both with the sender<span class="string">'s IP address are part from same subnet on this interface 3 - do not reply for local addresses configured with scope host, only resolutions for global and link addresses are replied 4-7 - reserved 8 - do not reply for all local addresses The max value from conf/&#123;all,interface&#125;/arp_ignore is used when ARP request is received on the &#123;interface&#125;</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;arp_ignore:定义对目标地址为本地IP的ARP询问不同的应答模式0 </p>
<ul>
<li>0 - (默认值): 回应任何网络接口上对任何本地IP地址的arp查询请求 </li>
<li>1 - 只回答目标IP地址是来访网络接口本地地址的ARP查询请求 </li>
<li>2 -只回答目标IP地址是来访网络接口本地地址的ARP查询请求,且来访IP必须在该网络接口的子网段内 </li>
<li>3 - 不回应该网络界面的arp请求，而只对设置的唯一和连接地址做出回应 </li>
<li>4-7 - 保留未使用 </li>
<li>8 -不回应所有（本地地址）的arp查询</li>
</ul>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;有关<strong>arp_announce</strong>的相关介绍：</p>
<blockquote>
<p>arp_announce - INTEGER Define different restriction levels for announcing the local source IP address from IP packets in ARP requests sent on interface: 0 - (default) Use any local address, configured on any interface 1 - Try to avoid local addresses that are not in the target’s subnet for this interface. This mode is useful when target hosts reachable via this interface require the source IP address in ARP requests to be part of their logical network configured on the receiving interface. When we generate the request we will check all our subnets that include the target IP and will preserve the source address if it is from such subnet. If there is no such subnet we select source address according to the rules for level 2. 2 - Always use the best local address for this target. In this mode we ignore the source address in the IP packet and try to select local address that we prefer for talks with the target host. Such local address is selected by looking for primary IP addresses on all our subnets on the outgoing interface that include the target IP address. If no suitable local address is found we select the first local address we have on the outgoing interface or on all other interfaces, with the hope we will receive reply for our request and even sometimes no matter the source IP address we announce. The max value from conf/{all,interface}/arp_announce is used. Increasing the restriction level gives more chance for receiving answer from the resolved target while decreasing the level announces more valid sender’s information.</p>
</blockquote>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;arp_announce:对网络接口上，本地IP地址的发出的，ARP回应，作出相应级别的限制: 确定不同程度的限制,宣布对来自本地源IP地址发出Arp请求的接口 </p>
<ul>
<li>0 - (默认) 在任意网络接口（eth0,eth1，lo）上的任何本地地址 </li>
<li>1 -尽量避免不在该网络接口子网段的本地地址做出arp回应. 当发起ARP请求的源IP地址是被设置应该经由路由达到此网络接口的时候很有用.此时会检查来访IP是否为所有接口上的子网段内ip之一.如果改来访IP不属于各个网络接口上的子网段内,那么将采用级别2的方式来进行处理. </li>
<li>2 - 对查询目标使用最适当的本地地址.在此模式下将忽略这个IP数据包的源地址并尝试选择与能与该地址通信的本地地址.首要是选择所有的网络接口的子网中外出访问子网中包含该目标IP地址的本地地址. 如果没有合适的地址被发现,将选择当前的发送网络接口或其他的有可能接受到该ARP回应的网络接口来进行发送.</li>
</ul>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;关于对arp_announce 理解的一点补充 </p>
<blockquote>
<p>Assume that a linux box X has three interfaces - eth0, eth1 and eth2. Each interface has an IP address IP0,<br>IP1 and IP2. When a local application tries to send an IP packet with IP0 through the eth2. Unfortunately,<br>the target node’s mac address is not resolved. Thelinux box X will send the ARP request to know<br>the mac address of the target(or the gateway). In this case what is the IP source address of the<br>“ARP request message”? The IP0- the IP source address of the transmitting IP or IP2 - the outgoing<br> interface? Until now(actually just 3 hours before) ARP request uses the IP address assigned to<br>the outgoing interface(IP2 in the above example) However the linux’s behavior is a little bit<br>different. Actually the selection of source address in ARP request is totally configurable<br>bythe proc variable “arp_announce”<br>If we want to use the IP2 not the IP0 in the ARP request, we should change the value to 1 or 2.<br>The default value is 0 - allow IP0 is used for ARP request.</p>
</blockquote>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;其实就是路由器的问题，因为路由器一般是动态学习ARP包的（一般动态配置DHCP的话），当内网的机器要发送一个到外部的ip包，那么它就会请求 路由器的Mac地址，发送一个arp请求，这个arp请求里面包括了自己的ip地址和Mac地址，而linux默认是使用ip的源ip地址作为arp里面 的源ip地址，而不是使用发送设备上面的 ，这样在lvs这样的架构下，所有发送包都是同一个VIP地址，那么arp请求就会包括VIP地址和设备 Mac，而路由器收到这个arp请求就会更新自己的arp缓存，这样就会造成ip欺骗了，VIP被抢夺，所以就会有问题。  </p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;arp缓存为什么会更新了，什么时候会更新呢，为了减少arp请求的次数，当主机接收到询问自己的arp请求的时候，就会把源ip和源Mac放入自 己的arp表里面，方便接下来的通讯。如果收到不是询问自己的包（arp是广播的，所有人都收到），就会丢掉，这样不会造成arp表里面无用数据太多导致 有用的记录被删除。<br>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;在设置参数的时候将arp_ignore 设置为1，意味着当别人的arp请求过来的时候，如果接收的设备上面没有这个ip，就不做出响应，默认是0，只要这台机器上面任何一个设备上面有这个ip，就响应arp请求，并发送mac地址</p>

	

	

</article>





	<span class="different-posts">📖 <a href="/page/35">more posts</a> 📖</span>



	</main>

	<footer class="footer">
	<div class="footer-content">
		
	      <div class="footer__element">
	<p>Hi there, <br />welcome to my Blog glad you found it. Have a look around, will you?</p>
</div>

	    
	      <div class="footer__element">
	<h5>Check out</h5>
	<ul class="footer-links">
		<li class="footer-links__link"><a href="/archives">Archive</a></li>
		
		  <li class="footer-links__link"><a href="/atom.xml">RSS</a></li>
	    
		<li class="footer-links__link"><a href="/about">about page</a></li>
		<li class="footer-links__link"><a href="/tags">Tags</a></li>
		<li class="footer-links__link"><a href="/categories">Categories</a></li>
	</ul>
</div>

	    

		<div class="footer-credit">
			<span>© 2017 失落的乐章 | Powered by <a href="https://hexo.io/">Hexo</a> | Theme <a href="https://github.com/HoverBaum/meilidu-hexo">MeiliDu</a></span>
		</div>

	</div>


</footer>



</body>

</html>
